Skip to content

fix(file-based): give each transferred file a unique staging path - #1078

Draft
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1785066655-unique-file-transfer-staging-paths
Draft

fix(file-based): give each transferred file a unique staging path#1078
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1785066655-unique-file-transfer-staging-paths

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

File transfer (use_file_transfer) staged every source file at {staging_dir}/{source_file_relative_path}. When two source files resolve to the same relative path, they share one staging file, which breaks syncs:

  1. Source downloads file A to /staging/files/Untitled document.docx, emits a record referencing it.
  2. Destination reads it, uploads it, then deletes the staging file (FileChunkTasklocalFile.delete()).
  3. Source downloads file B (a different source file) to the same path and emits another record.
  4. Whichever side loses the race fails: the destination hits java.io.FileNotFoundException: /staging/files/Untitled document.docx (No such file or directory), or the destination reads bytes of B for record A.

This is easy to hit on Google Drive (duplicate document names are allowed, and exported Google-native docs are named {name}.{ext}) and on any source when preserve_directory_structure is disabled and the same file name exists in several folders.

Fix: namespace the local staging path under a unique subdirectory, leaving source_file_relative_path (and therefore the destination object key) unchanged:

-local_file_path = path.join(staging_directory, file_relative_path)
+local_file_path = path.join(staging_directory, uuid.uuid4().hex, file_relative_path)

Because LOCAL_FILE_PATH comes from AbstractFileBasedStreamReader._get_file_transfer_paths, this covers the base upload() as well as the readers that call the helper themselves (source-google-drive, source-s3, source-microsoft-sharepoint) with no connector-side change.

The Check: source-google-drive CI job here fails on test_upload_file, which asserts the exact staging path; the test-only update it needs is in airbytehq/airbyte#82754 (same for source-microsoft-sharepoint; source-s3 already asserts with endswith). That check should go green once the monorepo PR merges.

Out of scope but worth flagging: duplicate relative paths still map to a single destination object key, so one file overwrites the other in the destination. That is inherent to path-mirroring naming and is not what makes the sync fail.

Resolves https://github.com/airbytehq/oncall/issues/12851:

Declarative-First Evaluation

Not applicable: the bug is in the Python CDK's file-based stream reader (staging path construction), not in a declarative connector manifest. No declarative feature can influence where the CDK stages a transferred file.

Test Coverage

  • New test_staging_paths_are_unique_per_file in unit_tests/sources/file_based/test_file_based_stream_reader.py asserts that two files resolving to the same relative path get identical file_relative_path but distinct local_file_path (parametrized for the duplicate-name and preserve_directory_structure: false cases). It fails on main (both calls return the same local path).
  • test_preserve_sub_directories_scenarios updated: it now asserts the local path is {staging_dir}/{unique_subdir}/{expected_relative_path} instead of a hardcoded path.
  • poetry run pytest unit_tests/sources/file_based -q → 796 passed, 1 skipped. ruff check, ruff format --check, and mypy pass.

Versioning

No connector version bump or changelog entry: this repo publishes from release notes, and pyproject.toml version is computed dynamically. Non-breaking — no schema, spec, stream, or state change; only the ephemeral local staging path changes, and it is communicated to the destination via staging_file_url in the same record.

Link to Devin session: https://app.devin.ai/sessions/8ef791c2602147a2951781e0af756828

Two source files that resolve to the same relative path shared one staging file, so the second download overwrote the first and the destination failed with FileNotFoundException after consuming and deleting the first copy.

Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1785066655-unique-file-transfer-staging-paths#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1785066655-unique-file-transfer-staging-paths

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

Copy link
Copy Markdown

PyTest Results (Fast)

4 135 tests  +2   4 123 ✅ +2   8m 2s ⏱️ +27s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 0388bfd. ± Comparison against base commit 7e96546.

@github-actions

Copy link
Copy Markdown

PyTest Results (Full)

4 138 tests  +2   4 126 ✅ +2   12m 3s ⏱️ -20s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 0388bfd. ± Comparison against base commit 7e96546.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

CI status on the two red connector checks:

  • Check: source-google-drive — real consequence of this change, and expected. test_upload_file asserts the exact staging path (assert expected_paths["staging_file_url"] in file_reference.staging_file_url), which no longer holds once the uuid segment is inserted. The test-only update is in test(source-google-drive, source-microsoft-sharepoint): allow unique staging subdirectory airbyte#82754 (also covers source-microsoft-sharepoint; source-s3 already asserts with endswith). Verified locally against this branch: google-drive 7 passed, sharepoint 48 passed, s3 26 passed. This check should go green once that PR merges.
  • Check: destination-motherduck — DockerHub connectivity, unrelated to this diff: failed to resolve source metadata for docker.io/docker/dockerfile:1: ... dial tcp 44.205.146.148:443: i/o timeout in test_docker_image_build_and_*. No CDK code involved; a re-run should clear it (I don't have permission to re-run jobs here).

All CDK checks — Pytest (all Python versions), Ruff, MyPy, package build — are green.


Devin session

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants